Skip to content

Feat/artifactory full masterminds's semver support and path templating#556

Merged
notrepo05 merged 2 commits intomainfrom
feat/artifactory-source-full-semver-support
Feb 24, 2026
Merged

Feat/artifactory full masterminds's semver support and path templating#556
notrepo05 merged 2 commits intomainfrom
feat/artifactory-source-full-semver-support

Conversation

@notrepo05
Copy link
Member

@notrepo05 notrepo05 commented May 1, 2025

Issue description

This PR aims to fix

  1. Downloading undesired / unrelated files
    e.g. bosh-releases/smoothie/9.9/mango/tgz-9.9mangosmoothie.9.9-- and some-release-1.2.3-notices.zip
  2. Not respecting the Kilnfile Path Template
  3. Ignoring pre-release semantics

I'm especially hoping to get feedback because it's a breaking change for the Artifactory Release Source: pre-release versions will no longer be found by the default * release constraint.
Breaking changes:
⚠️ Instead the constraint >0-0 would be needed
⚠️ It drops support for 4 segment versions like 1.2.3.4.
ℹ️ the v prefix is allowed
ℹ️ when stemcells are usedfind-release-version requires a stemcell version to be set

The reason for requiring this breaking change is that the resource has been matching only major.minor.patch.fourth and incidentally picking up / ignoring the pre-release segment. But I’d like it to fully embrace pre-release support by swapping the major.minor.patch regex for one that captures both pre-release (1.2.3-pre.1) and build metadata (1.2.3+build.1). For example, we’ll no longer see the remote file some-release-1.2.3-dev.1.tgz updating the Kilnfile.lock to 1.2.3, but rather 1.2.3-dev.1. If we default '*' to include pre-releases then there isn't a clear way to exclude them.

More details

Currently, the Artifactory Release Source is extracting versions matching the following the pattern ([-v])\d+(.\d+)*. The first match is assumed to be the bosh release and if a second exists it's the stemcell's version. The Kilnfile template is partially ignored and order takes precedent here. Also, since we only match on e.g. 1.2.3and not 1.2.3-build.1 this implies there's no pre-release sorting beyond creation date from Artifactory to the best of my knowledge. Since the Kilnfile template isn't strictly enforced files you get the examples seen under "Downloading undesired / unrelated files" at the top of this comment.

Proposal

File parsing

I'd like to propose

  1. the artifactory source's find-release-version fully utilizes mastermind's semantics (used by other kiln release sources)
  2. that Kilnfile templates are used as the basis for deciding if a remote Artifactory file is a match. This template can be turned into a path pattern with a version glob and translated into an AQL query for Artifactory, which is the approach.

For example, among others, these templates have been tested:

  • release-version-beginning-path/{{.StemcellVersion}}/{{.Version}}/{{.Name}}/{{.Name}}.tgz <-- this would have failed
  • release-version-in-path/{{.Name}}/{{.StemcellOS}}/{{.StemcellVersion}}/{{.Version}}/{{.Name}}.tgz

The flow is basically

    path_template: release-version-beginning-path/{{.StemcellVersion}}/{{.Version}}/{{.Name}}/{{.Name}}.tgz

turns into an AQL query

        "items.find": {
	  "repo": "tas-ecosystem-generic-dev-local",
          "$and": [
            { "path": { "$match": "release-version-beginning-path/1.1.1-build.1/*/release-name" } },
            { "name": { "$match": "release-name-*.tgz" } }
          ]
        }

whose results are parsed by a regex version of the path template

release-version-beginning-path/<semver regex>/<semver regex>/release-name

to extract the version(s), check against mastermind's version constraint, and sort for latest.

Using AQL

AQL (Artifactory Query Language) supports searching for files using the path template globbing scheme described above. So this PR moves file searching responsibility to jfrog-client-go. This also updates checksums without requiring download

Semver semantics

Adopting mastermind's semantics changes the meaning of * for the artifactory source's find-release-version since it won't match pre-releases, making this a breaking change. But it helps tiles avoid accidental ingestion of unwanted files.while still allowing pre-release matches by opting in with a constraint like >0-0.

See https://github.com/Masterminds/semver?tab=readme-ov-file#working-with-prerelease-versions

Notes

  • This will support pre-release and build metadata extensions in the versions. Both are extensions to semantic versions as described here https://semver.org/. For example 1.2.3+build.1 and 1.2.3-rc.1 will be supported.
  • Jfrog's library is clunky to use for file mutation so I'm excluding the migration to it for the mutating commands (upload/download e.g.). This should be a follow-on

@notrepo05
Copy link
Member Author

This is intended to supersede #533 which was reverted to maintain compatibility with pre release versions

@notrepo05 notrepo05 force-pushed the feat/artifactory-source-full-semver-support branch from 7b52c03 to d391e7e Compare May 1, 2025 23:33
@notrepo05
Copy link
Member Author

Notes:

  • The leading v isn't currently removed from remotely discovered versions, but this doesn't impact finding new versions. I need to work out what to do in this case
  • When a stemcell os is configured but the version isn't it's treated as an error.
  • This will require tiles using pre-releases to adopt the new pre-release constraint

@notrepo05 notrepo05 force-pushed the feat/artifactory-source-full-semver-support branch 2 times, most recently from 879dba6 to eedecf8 Compare May 2, 2025 20:59
@notrepo05 notrepo05 changed the title Feat/artifactory source full semver support Feat/artifactory full masterminds's semver support May 2, 2025
@notrepo05
Copy link
Member Author

I ran git bisect and discovered that 3b4fd72 might be breaking the bake acceptance tests. I've reverted it for now in order to have a clean test foundation

@notrepo05
Copy link
Member Author

Tests pass otherwise, so reapplying with that known. That should be followed up on in a different PR :)

@mamachanko
Copy link
Contributor

I ran git bisect and discovered that 3b4fd72 might be breaking the bake acceptance tests. I've reverted it for now in order to have a clean test foundation

#559 captures the regression and a fix

@notrepo05 notrepo05 requested a review from dlinsley May 7, 2025 09:19
@notrepo05 notrepo05 changed the title Feat/artifactory full masterminds's semver support Feat/artifactory full masterminds's semver support and path templating May 7, 2025
@notrepo05 notrepo05 requested a review from rizwanreza May 7, 2025 09:29
@notrepo05 notrepo05 force-pushed the feat/artifactory-source-full-semver-support branch 3 times, most recently from 15a348f to 3741c49 Compare February 17, 2026 20:50
@notrepo05 notrepo05 marked this pull request as ready for review February 17, 2026 20:56
@notrepo05 notrepo05 requested a review from davewalter February 24, 2026 02:03
for exact matches and proper pre-release version support.
@notrepo05 notrepo05 force-pushed the feat/artifactory-source-full-semver-support branch from 0801978 to a3b2c44 Compare February 24, 2026 02:07
@notrepo05 notrepo05 merged commit 9b1b8b7 into main Feb 24, 2026
3 checks passed
@notrepo05 notrepo05 deleted the feat/artifactory-source-full-semver-support branch February 24, 2026 02:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants